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Abstract 

This paper presents an approach to the problem of binary 
classification using ensemble neural networks based on in- 
terval neutrosophic sets and bagging technique. Each com- 
ponent in the ensemble consists of a pair of neural networks 
trained to predict the degree of truth and false membership 
values. Uncertainties in the prediction are also estimated 
and represented using the indeterminacy membership val- 
ues. These three membership values collectively form an 
interval neutrosophic set. In order to combine and classify 
outputs from components in the ensemble, the outputs of an 
ensemble are dynamically weighted and summed. The pro- 
posed approach has been tested with three benchmarking 
U Cl data sets, which are ionosphere, pima, and liver. The 
proposed ensemble method improves the classification per- 
formance as compared to the simple majority vote and av- 
eraging methods which were applied only to the truth mem- 
bership value. Furthermore, the results obtained from the 
proposed ensemble method also outperform the results ob- 
tained from a single pair of networks and the results ob- 
tained from a single truth network. 



1. Introduction 

In order to solve the problem of classification and predic- 
tion, an ensemble of accurate and diverse neural networks 
was found capable of providing better results than a sin- 
gle neural network [4]. In the normal process of utilizing 
neural network ensemble, each network component has to 
be trained and then the outputs obtained from all networks 
in the ensemble are combined. However, there are situa- 
tions that outputs from the networks are differed from one 
another. Dietterich [2] suggested that if two classifiers pro- 
duce different errors on new input data then both classifiers 
are considered to be “diverse”. Diversity in an ensemble 
of neural networks can be handled by manipulating input 
data or output data. An example of algorithm that manipu- 
lates the diversity using output data is error-correcting out- 



put coding. Bagging and boosting are examples of such 
algorithms that manipulate diversity using input data. In 
the error-correcting output coding algorithm, a unique code- 
word which is a binary string of length n, is created for each 
class and used as distributed output representation [3]. Bag- 
ging provides diversity by randomly resampling the original 
training data into several training sets [1] whereas boosting 
provides diversity by manipulating each training set accord- 
ing to the performance of the previous classifier [8]. Fur- 
thermore, manipulating the input features can also provide 
diversity in the ensemble [2]. In addition, diversity can be 
provided by applying artificial training samples. Melville 
and Mooney [6] built a training set for each new classifier by 
adding artificially constructed samples to the original train- 
ing data. In order to construct sample labels, they assigned 
the class label that disagrees with the current ensemble to 
the constructed sample label. 

In this paper, ensemble diversity is created using bagging 
algorithm. Bagging is created based on bootstrap resam- 
pling. Each training set in an ensemble is generated from 
the original training data using random resampling with re- 
placement. Each generated training set contains the same 
number of training patterns as the original training set. The 
outputs of the ensemble can be aggregated using averaging 
or majority vote. Combination of the outputs in the ensem- 
ble can improve the accuracy results. However, uncertainty 
still exists. In this paper, we apply interval neutrosophic 
sets [9] in order to represent uncertainty in the prediction. 

This research follows the definition of interval neutro- 
sophic sets defined by Wang et al. [9]. The membership 
of an element to the interval neutrosophic set is expressed 
by three values: truth membership, indeterminacy member- 
ship, and false membership. The three memberships are 
independent although in some special cases, they can be 
dependent. In this study, the indeterminacy membership 
depends on both truth and false memberships. The three 
memberships can be any real sub-unitary subsets and can 
represent imprecise, incomplete, inconsistent, and uncer- 
tain information. In this paper, the memberships are used 
to represent uncertainty information. For example, let A 
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Figure 1. The proposed training model based on the integration of interval neutrosophic sets with 
bagging neural networks. 



be an interval neutrosophic set, then x(75, {25, 35, 40}, 45) 
belongs to A means that x is in A to degree of 75%, x is un- 
certain to degrees of 25% or 35% or 40%, and x is not in A 
to degree of 45%. The definition of an interval neutrosophic 
set is described below. 

Let X be space of points (objects). An interval neutro- 
sophic set in X is defined as: 

A = {x(T a (x),Ia(x),F a (x))\x £ X A 
T a :X—>[ 0,1] A 
I a : X — ► [0, 1] A ; 

F a : X — - [0, 1]} 

where 

Ta is the truth membership function, 

I A is the indeterminacy membership function. 

Fa is the false membership function. 

In this paper, we create a pair of neural networks for each 
component in the ensemble. In each pair, two networks are 
opposite to each other. Both neural networks are trained 
with the same bag of data but disagree in the output tar- 
gets. The first network predicts degrees of truth member- 
ship whereas the second network predicts degrees of false 
membership. The predicted outputs from both networks are 
supposed to be complement to each other. However, both 
predicted outputs may not completely complement to each 
other. Uncertainty may occur in the prediction. In this 
study, we represent uncertainty in the form of indetermi- 
nacy membership value. The three memberships form an 
interval neutrosophic set and are used for decision making 
in the binary classification. 

The rest of this paper is organized as follows. Section 2 
explains the proposed method for the binary classification 
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with the assessment of uncertainty using interval neutro- 
sophic sets and bagging. Section 3 describes the data set 
and the results of our experiments. Conclusions and future 
work are presented in Section 4. 

2. Binary classification using interval neutro- 
sophic sets, ensemble neural network, and 
bagging 

In our previous papers [5], we integrated neural networks 
with interval neutrosophic sets in order to classify mineral 
prospectivity into deposit or barren cell. A pair of neural 
networks was created to predict degree of truth and false 
membership values. The predicted truth and false member- 
ship values were then compared to give us the classification 
results. Uncertainties in the classification were calculated 
as the difference between the truth and false membership 
values and were represented using indeterminacy member- 
ship values. We found that interval neutrosophic sets can 
represent uncertainty information and support the classifica- 
tion quite well. In this paper, we extend the work from our 
previous paper [5] by applying ensemble neural networks, 
interval neutrosophic sets, and a bagging technique to the 
problem of binary classification. Figure 1 shows the pro- 
posed training model that applies interval neutrosophic sets 
and a bagging technique to the ensemble neural network. 
Each component in the ensemble consists of a pair of neu- 
ral networks, which are the truth neural network (Truth NN) 
and the falsity neural network (Falsity NN). The truth net- 
work is trained to predict degrees of truth membership. The 
falsity network is trained to predict degrees of false mem- 
bership. Both networks are based on the same architecture. 
They apply the same bag for training. The difference be- 
tween both networks is that the falsity network is trained 
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with the complement of the target output values presented 
to the truth network. In the training phase, each bag of data 
presented to each component in the ensemble is created us- 
ing bootstrap resampling. In this study, each bootstrap sam- 
ple or bag of data is created by random selection of input 
patterns from the training data set with replacement. Each 
bag contains the same number of training patterns as the 
original data set. Hence, m bags of data are applied to m 
pairs of truth and falsity neural networks. 

In the test phase, the test data is applied to each com- 
ponent in the ensemble. From our testing model, each pair 
of the truth and falsity networks predict n pairs of the truth 
and false membership values where n is the total number of 
patterns. For each pair of the truth and false membership 
value, the truth membership value is supposed to be com- 
plement to the false membership value. For example, if the 
truth membership value is 1 , the false membership value is 
supposed to be 0. If the difference between these two val- 
ues is 1 then the uncertainty value will be 0. However, the 
predicted truth membership value is not necessary to be one 
hundred percent complement to the predicted false mem- 
bership value. Uncertainty may occur in the prediction. For 
instance, if the truth membership value is 0.5 and the false 
membership value is also a value of 0.5 then the uncertainty 
value will be 1 . Consequently, we compute the uncertainty 
value as the difference between the truth membership and 
false membership values. If the difference between these 
two values is high then the uncertainty is low. On the other 
hand, if the difference is low then the uncertainty value 
is high. Figure 2 shows the relationships among the truth 
membership, false membership, and uncertainty values. In 
this paper, we represent uncertainty value in the form of in- 
determinacy membership value. Hence, the output obtained 
from each component is represented as an interval neutro- 
sophic set. The three memberships created from each com- 
ponent can be defined as the following. 

Fet Xj be the j'-th output at the j-th component, where 
j = 1, 2, 3, ..., m. Fet Aj be an interval neutrosophic set in 
Xj . Aj can be defined as 




Figure 2. Relationships among the truth 
membership value, false membership value, 
and uncertainty value. 



are dynamically weighted average. Also, the false member- 
ship values obtained from all components are dynamically 
weighted average. After that, the average truth membership 
and the average false membership values are compared in 
order to classify the input pattern into a binary class. In 
this study, the weight is dynamically created based on the 
indeterminacy membership value. The more weight means 
the more certainty in the prediction. We calculate the cer- 
tainty as the complement of the indeterminacy membership 
value. Fet P(xi ) be an average truth membership value 
based on weights. Fet Q(xi) be an average false member- 
ship value based on weights. Fet Wj(x ,) be the weight 
based on the indeterminacy membership value at the j- 
th component. P, Q and W can be defined as the following. 



P(Xi) = x T \ C )) (4) 

3 = 1 



Aj = {x(T Aj (x),I Aj (x),F Aj (x)) I x e Xj A 
T Aj : Xj ► [0, 1] A 
I Aj : Xj — * [0, 1] A 
F Aj : Xj — * [0, 1]}, 

I Aj {x) = l-\T Aj {x)-F Aj {x)\, (3) 



Q{xi) = ^2(Wj(x i) x F Aj (xi)) 

3 = 1 



Wj( Xi ) 



1 - I Aj (Xi) 
£”ll(l - T Aj( x i)) 



( 5 ) 

( 6 ) 



where 

TV is the truth membership function, 

I Aj is the indeterminacy membership function, 

F Aj is the false membership function. 

In order to combine the outputs obtained from all com- 
ponents for each input pattern, the truth membership values 



After the average truth membership and the average false 
membership values are computed for each input pattern, 
these two values are compared. If the average truth mem- 
bership value is greater than the average false membership 
value ( P(xi ) > Q(x-i )) then the input pattern is classified 
as a value 1 . Otherwise, the input pattern is classified as a 
value 0. 
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3. Experiments 

3.1. Data set 

Three data sets from UCI Repository of machine learn- 
ing [7] are used for binary classification in this paper. Ta- 
ble 1 shows the characteristics of these three data sets. The 
size of training and testing data used in our experiments are 
also shown in this table. 



Table 1. Data sets used in this study. 



Name 


ionosphere 


pima 


liver 


No. of Class 


2 


2 


2 


No. of Feature 


34 


8 


6 


Feature Type 


numeric 


numeric 


numeric 


Size of Samples 


351 


768 


345 


Size of Training Data 


200 


576 


276 


Size of Test Data 


151 


192 


69 



3.2. Experimental methodology and results 

In this experiment, three data sets named ionosphere, 
pima, and liver from UCI Machine Learning Repository are 
applied to our model. Each data set is split into a training 
set and a testing set. The sizes of both sets are shown in 
table 1. For each training data set, thirty bags are created 
using bootstrap resampling with replacement and applied to 
thirty components in the ensemble. For each component, 
a pair of feed-forward backpropagation neural networks is 
trained in order to predict degree of truth membership and 
degree of false membership values. In this paper, we want 
to focus on our technique that aims to increase diversity by 
creating a pair of opposite networks in each component in 
the ensemble. Therefore, all networks in each ensemble ap- 
ply the same parameter values and are initialized with the 
same random weights. The only difference for each pair of 
networks is that the target outputs of the falsity network are 
equal to the complement of the target outputs used to train 
the truth network. In the ionosphere data set, all networks 
in the ensemble have the same architecture which composes 
of thirty-four input units, a single output unit, and one hid- 
den layer constituting of sixty-eight neurons. In the pima 
data set, all networks compose of eight input units, a sin- 
gle output unit, and one hidden layer constituting of sixteen 
neurons. In the liver data set, all networks compose of six 
input units, a single output unit, and one hidden layer con- 
stituting of twelve neurons. 

In the test phase, after both truth and false membership 
values are predicted, the indeterminacy memberships are 
then computed using an equation 3. In order to combine the 
output from the networks within the ensemble, we apply our 



Table 2. Average classification results for the 
test data set obtained by applying the pro- 
posed methods and the existing methods 





Technique 


Ionosphere 

%correct 


Pima 

%correct 


Liver 

%correct 


Bagging 


dynamically 

weighted 

average 


98.21 


77.97 


74.13 


Simple 

averaging 


98.01 


77.16 


69.93 


Simple 
majority vote 


98.15 


76.38 


71.30 


single 


Tj > Fj 


96.42 


74.74 


66.52 


T.j > 0.5 


93.54 


70.49 


62.68 



technique described in the previous section. In this paper, 
we do not consider the optimization of the prediction but 
concentrate only on the improvement of the prediction. In 
the experiment, we try twenty ensembles for each UCI data 
set. Each ensemble includes thirty different bags of training 
set. For each data set, the classification accuracy results ob- 
tained from all twenty ensembles are averaged and shown 
in Table 2. Furthermore, we also compare the average re- 
sults obtained from our bagging technique (row 2) among 
the average results obtained from the existing bagging tech- 
niques (row 3-4), the existing technique using a single pair 
of networks (row 5), and the existing technique using only 
a single truth neural network (row 6). 

In the third row of Table 2, the results obtained from the 
simple averaging technique are shown. In this technique, 
only the truth neural network constitutes each component 
in an ensemble. The truth membership values obtained 
from all components are averaged and then compared to the 
threshold value of 0.5. If the average result is greater than 
the threshold value then the input pattern is classified as a 
value 1 . Otherwise it is classified as a value 0. In this tech- 
nique, twenty ensembles are created for each data set, and 
the average results are shown. 

In the simple majority vote technique, only the truth 
neural networks constitute an ensemble. The truth mem- 
bership value obtained from each network is compared to 
the threshold value of 0.5. If the truth membership value 
is greater than the threshold value then the input pattern is 
classified as a value 1 . Otherwise it is classified as a value 
0. After that, all results are voted for each input pattern. If 
at least half of the results yield a value 1 then the input pat- 
tern is classified as a value 1 . Otherwise, it is classified as a 
value 0. In this technique, twenty ensembles are created for 
each data set. The average results are shown in the fourth 
row. 
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Table 3. Total number of correct and incorrect 
outputs predicted from the proposed tech- 
nique for the test set of pima data. 



Uncertainty 


Number of cell 


%correct 


value 


level 


correct 


incorrect 


0.6940-0.9998 


High 


21 


17 


55.26 


0.3881-0.6939 


Med 


52 


19 


73.24 


0.0821-0.3880 


Low 


79 


4 


95.18 



In the fifth row, the technique presented in our previous 
paper [5] is applied. A single pair of neural networks is 
trained. After that, the predicted truth and false membership 
values are compared in order to classify the binary class. If 
the truth membership value is greater than the false mem- 
bership value then the input pattern is classified as a value 
1 . Otherwise, it is classified as a value 0. In this technique, 
we try twenty pairs of neural networks with twenty differ- 
ent randomized training sets for each data set. All twenty 
results are then averaged. The average result belonging to 
each data set is shown in Table 2. 

In the last row, a single neural network is trained in order 
to provide the truth membership value. The output of the 
network is then compared to the threshold value of 0.5. The 
input pattern is assigned a value 1, if the output is greater 
than the threshold value. Otherwise it is assigned a value 0. 
Similar to the previous techniques, twenty neural networks 
are trained with twenty different randomized training sets. 
All twenty predicted results are averaged. The average re- 
sults are shown in Table 2. 

From the experiments, we found that the technique of the 
comparison between the truth and false membership values 
gives us better performance compared to the technique us- 
ing the threshold value for the classification. We also found 
that the bagging technique improves the classification per- 
formance as compared to the technique that applies only 
a single pair of opposite networks or only a single network. 
Furthermore, our experiments show that the results obtained 
from the proposed ensemble technique (row 2) outperform 
the results obtained from the other existing techniques used 
in this paper. 

In addition, our approach has an ability to represent un- 
certainty in the classification. For each input pattern, uncer- 
tainty in the classification can be calculated as the difference 
between P(x) and <3(x), which are the average truth mem- 
bership value and the average false membership value based 
on weights, respectively. This value can be used to support 
the confidence in the classification. For example, table 3 
shows the ranges of uncertainty in the classification of pima 
data set. Uncertainty values are categorized into three lev- 
els: High, Med, and Low. This table represents the total 



number of correct and incorrect outputs predicted from the 
proposed ensemble technique. The table shows that most of 
the outputs that have low level of uncertainty are correctly 
classified. Hence, this uncertainty level can be used as an 
indicator in order to support the decision making. 

4. Conclusion and future work 

This paper has applied a pair of opposite neural networks 
to input patterns derived from bagging technique for the 
prediction of the truth membership and false membership 
values. A pair of networks constitutes a component in the 
ensemble. The difference between each pair of the truth and 
false membership values gives us an uncertainty value or an 
indeterminacy membership value. The three memberships 
form an interval neutrosophic set and are used for dynami- 
cally weighted averaging. The advantage of our approach 
over a simple averaging and majority vote approaches is 
that the indeterminacy membership values provide an es- 
timate of the uncertainty of the classification. In addition, 
our experimental results indicate that our proposed ensem- 
ble technique improves the classification performance com- 
pared to the existing techniques. In the future, we will apply 
our technique to the problem of multiclass classification. 
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